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The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to 
breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


The EEF aims to raise the attainment of children facing disadvantage by: 


e Identifying promising educational innovations that address the needs of disadvantaged children 
in primary and secondary schools in England; 

e Evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; 

e Encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 


The EEF was established in 2011 by the Sutton Trust, as lead charity in partnership with Impetus Trust 
(now part of Impetus — The Private Equity Foundation) and received a founding £125m grant from the 
Department for Education. 


Together, the EEF and Sutton Trust are the government-designated What Works centre for improving 
education outcomes for school-aged children. 
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Executive summary 


The project 


The Changing Mindsets project sought to improve academic attainment by supporting pupils to develop 
a growth mindset: the belief that intelligence is not a fixed characteristic and can be increased through 
effort. Previous research (Good et al/., 2003; Blackwell et a/., 2007) has suggested that holding this 
belief enables pupils to work harder and achieve better results. The project consisted of two separate 
interventions: 


e an intervention that taught pupils directly about the malleability of intelligence through six 
workshops, which were delivered by undergraduates from the University of Portsmouth, and 
four further sessions delivered by two local organisations: the Education Business Partnership, 
and Pompey Study Centre (now called Portsmouth in the Community). 

e a professional development course that trained teachers on approaches to developing and 
reinforcing growth mindsets through their teaching. This course consisted of two half days of 
instruction. 


The project targeted Year 5 pupils in Portsmouth, Southampton and Hampshire. The delivery of the 
interventions was led by the University of Portsmouth and took place between January and May 2013. 


Key conclusions 


1. Pupils who received the growth mindset workshops made an average of two additional months’ 
progress in English and maths. These findings were not statistically significant, which means that 
we cannot be confident that they did not occur by chance. However, the finding for English was 
close to statistical significance, and this suggests evidence of promise. 

2. Pupils whose teachers received the professional development intervention made no additional 
progress in maths compared to pupils in the control group. These pupils made less progress in 
English than the control group, but this finding is not statistically significant and we cannot be sure 
that it did not occur by chance. 

3. FSM-eligible pupils who were involved in the professional development intervention gained a better 
understanding of the malleability of intelligence. 

4. Intervention and control schools were already using some aspects of the growth mindsets 
approach. This may have weakened any impact of the interventions. 


5, Future trials could examine the impact of a programme that combines the two interventions and 
runs for a longer period of time. 
How secure is this finding? Security rating awarded as part of 
the EEF peer review process 


The pupil workshop intervention was evaluated using a randomised controlled trial involving six schools 
and 286 pupils. The findings from this evaluation have moderate to low security. Pupils were randomly 
allocated to receive either the intervention or to an active control group where they received the same 
amount of extra support with study skills, but without the focus on developing a growth mindset. This 
comparison allows us to isolate the impact of the focus on growth mindsets and conclude that any 
impact was not caused simply by pupils receiving extra time and attention. The security of the trial was 
weakened when one of the six recruited schools withdrew from the project as this reduced the number 
of pupils in the analysis. 


The teacher training intervention was evaluated using a randomised controlled trial involving 30 schools 
and 1,505 pupils. The findings from this evaluation have moderate security. Schools were randomly 
allocated to receive either the intervention or a ‘business-as-usual’ control group, where teachers did 
not receive any extra training as a result of participation in the trial. The security of these findings was 
limited by evidence that the intervention and control groups had different numbers of pupils eligible for 
free school meals and different average levels of prior attainment. This introduces the risk that any 
difference in outcomes between the two groups is caused by the different composition of the groups, 
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not by the impact of the intervention. The evaluator attempted to control for these differences using 
statistical techniques in the final analysis. 


There was some evidence from the process evaluation that teachers in the control groups of both trials 
were already aware of growth mindsets theory and using it to inform their practice. This previous 
exposure to the approach may have weakened the relative impact of the intervention. 


Findings 


Pupils who received the growth mindset workshops made an average of two additional months’ 
progress in both English and maths compared to those in the control group. These findings were not 
statistically significant, which means that we cannot be confident that they did not occur by chance. 
However, the finding for English was close to statistical significance, and this suggests evidence of 
promise. Pupils whose teachers received the professional development intervention made no additional 
progress in maths compared to pupils in the control group. These pupils made less progress in English 
than pupils in the control group, but this finding was not statistically significant and we cannot be sure 
that it did not occur by chance. 


The evaluation also measured the impact of the interventions on pupils’ theories of intelligence, using 
measurements created by Dweck (1999). Pupils involved in both interventions achieved higher scores 
on these measures than those in the control group, but the security of these findings is low and it is 
possible that they occurred by chance. The professional development intervention led to higher scores 
on the growth mindset measurements for pupils eligible for FSM and this finding is secure. 


Previous research from the US has suggested that growth mindset interventions can have a positive 
impact on attainment. Good et al. (2003) found that using university students to teach pupils about the 
malleability of intelligence led to large improvements in standardised tests. 


How much does it cost? 


The estimated cost of implementing the pupil workshops intervention over three years is £397 per pupil 
per year. The estimated cost of implementing the teacher training intervention over three years is £16 
per pupil per year (£678 per school per year). Over the single year of this evaluation the teacher training 
intervention cost £49 per pupil (£2,035 per school). Teachers required two half days of supply cover to 
participate in the teacher training, which schools might choose to deal with in ways that may or may not 
incur a financial cost. 
Group Effect Additional Security Cost 
size months’ relat ale} ee lalale} 
progress 


Teacher training (maths, all pupils) 0.01 Omonths @@8 £ 
Teacher training (English, all pupils) -0.11 -2month Q@88 £ 


Teacher training (maths, FSM pupils) 0.04 +1 month? 


Pupil workshops (intervention (maths, all 0.1 +2months @@ ££E 
pupils) 
Pupil workshops (English, all pupils) 0.18 +2months @@ ELLE 


Pupil workshops intervention (maths, FSM 0.11 +2 months 
pupils) 
Pupil workshops (English, FSM pupils) 0.17 +2 months 


1 Since this report was published, the conversion from effect size into months of additional progress has been slightly revised. If 
this result was reported using the new conversion, it would be reported as 0 months of additional progress rather than +1. See 
here for more details. 
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Introduction 


The Changing Mindsets project was developed by Growing Learners, a group of education research 
psychologists, led by Dr Sherria Hoskins and based in the Psychology Department at the University of 
Portsmouth. The Growing Learners team work to support schools to improve their pupils’ expectations 
and attainment, using evidence-based practice to help them to become resilient, confident and effective 
learners. The services and products that they offer are underpinned by psychological and educational 
theory and research, including research conducted by the team themselves. 


The Changing Mindset intervention was based on the theory of implicit theories of intelligence. Carol 
Dweck (1999) argued that it was not ability or belief in that ability that predicts resilience and 
perseverance in the face of challenge and failure, it is the individual’s belief about the nature of ability 
(referred to as ‘self theory of intelligence’, also known as ‘mindset’). 


Two types of intelligence belief were identified by growth mindset theory: incremental theories (growth 
mindsets) or entity theories (fixed mindsets). Individuals with a growth mindset believe that they can 
develop their intelligence, while individuals with a fixed mindset believe that their intelligence is innate 
(i.e. “I was born this way/this is what | am’). 


Dweck argues that teachers can help children to develop a growth mindset by praising them for their 
effort and persistence, rather than their innate intelligence. For example, if a teacher wanted to praise 
a pupil for a piece of work, they should say “well done for working really hard” rather than “well done, 
you must be really smart.” 


Intervention 


This evaluation was conducted on two separate, but similar, interventions. The first intervention 
consisted of a series of six workshops for pupils. The workshops were delivered in schools by university 
students employed as project support assistants (PSAs). Half of the pupils on the course were randomly 
allocated to receive the mindset intervention, while the other half received general study support. The 
workshops taught pupils about the malleable nature of intelligence, and sought to help pupils develop 
a growth mindset. The workshops also used hands-on work in maths and literacy—demonstrating how 
not giving up and exploring different strategies can help to find solutions. This part of the intervention 
was designed and managed by the University of Portsmouth. As far as possible, the format of the course 
was standardised across schools. 


The second intervention consisted of staff from the University of Portsmouth training teachers to teach 
pupils that intelligence is not a fixed characteristic, and to reinforce this in their lessons through the way 
they communicate with pupils. This was delivered through two half-days of teacher training. 


Background evidence 


This work builds on the pioneering research by Carol Dweck (1999) and her colleagues about the 
theories that children hold about their intelligence -, that is, whether it is a ‘fixed entity’ or a ‘malleable 
quality’ that can be developed. Importantly, having a growth mindset is seen to create resilience when 
faced with failure or difficulties in learning. A growth mindset sends a child positive messages about 
effort and strategy, leading pupils to try harder, or to try a different strategy for learning. In comparison, 
for children with a fixed mindset failure will send a negative message about who they are. This, it is 
argued, will damage their self-esteem and is likely to lead them to avoid the same task in the future. 


These concepts were first identified by Seligman (Seligman et al/., 1968) as the perception that an 
obstacle is impossible to overcome, termed ‘learned helplessness’. In contrast, individuals who are 
‘mastery oriented’ use failure to motivate themselves; they are resilient to challenge and failure (Dweck, 
1999; Blackwell et al., 2007). 
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A key study (Blackwell et al., 2007) found that 7th grade students in America who agreed with the idea 
that ‘you can always change how intelligent you are’ outperformed similar peers in the same school 
who believed that ‘you have a certain amount of intelligence and you can’t do much to change it’, and 
the gap in performance grew over time. Another US study (Good et a/., 2003) found that using university 
student mentors to teach pupils about their ability to grow their intelligence led to large improvements 
in standardised tests. Kornilova et al. (2009) also found that mindset is a greater predictor of academic 
performance than intelligence (as measured by IQ). 


In the UK there is growing interest among policymakers in promoting character and resilience education 
in schools, which encompasses the growth mindset approach. For example, the Character Education 
Grant Fund was established in January 2015 by the Department of Education to help schools ensure 
that more children develop a set of character traits, attributes and behaviours that underpin success in 
education and work. Despite growing interest in mindset theory and approaches, we are unaware of 
any rigorous trials assessing the impact of the growth mindset approach in the UK. This evaluation was 
set up as an efficacy trial, meaning that it sought to test whether the intervention can work under ideal 
or developer-led conditions. 


Evaluation objectives 
The evaluation was designed to assess the impact of: 


e a pupil workshop intervention—six weeks of workshops focused on growth mindsets. This 
intervention was compared with an active control group where pupils received the same amount 
of workshops but with a focus on general study skills rather than developing a growth mindset. 

e an INSET intervention—two sessions of teacher training in how to teach pupils about the 
malleability of their intelligence and how to reinforce this in lessons. This intervention was, 
compared with a ‘business as usual’ control group where teachers didn’t receive any additional 
training. 


The evaluation aimed to explore any differential impacts for pupils eligible for free school meals (FSM) 
and pupils with low scores on a pre-test. The evaluation initially aimed to explore whether the 
interventions had different impacts on different ethnicities, but data on pupils’ ethnicity was not made 
available to the research team. 


The objective of the process evaluation was to examine implementation and fidelity to the intervention. 
It also aimed to identify factors that affect the impact of the Changing Mindsets interventions and might 
explain the findings of the quantitative evaluation. We aimed to look for evidence of effectiveness and 
issues which would need to be considered for a wider roll-out of Changing Mindsets, whether or not it 
was found to have an effect on learning. 


Project team 


The University of Portsmouth Psychology Department delivered both interventions. The Education 
Department of Portsmouth Local Authority was involved in recruiting schools and assisted in the 
evaluation by facilitating access to schools by evaluators. Undergraduate students were employed as 
project support assistants (PSAs) to deliver the workshops in the pupil intervention. They were recruited 
through a competitive process and trained to deliver the workshops by the academic team at the 
University of Portsmouth. The project leaders at the University of Portsmouth led the teacher training 
for the INSET intervention. The PSAs also collected all pupil assessment data required by the 
evaluation. They were not aware of what type of workshop the school pupils had received, at all points 
of data collection. NIESR led the independent evaluation, agreed the research design, randomised the 
schools and pupils, and carried out the impact analysis and process study. 


Ethical review 


The University of Portsmouth and the Education Department of Portsmouth Local Authority were 
responsible for recruiting the schools. They made the individual contact with each school and 
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provided a full explanation of the evaluation during the scheduled set-up meetings. Headteachers 
were asked to give signed consent for their school to take part (see appendices 1 and 2). In signing 
up for the project, schools were fully aware that they were giving consent for the evaluation to take 
place and what this would involve. 


At the onset of the project (December 2012), parents were sent an information letter about the project. 
They were subsequently sent an opt-out letter regarding data sharing (See Appendix 3) explaining 
linkage to the pupil data held in the National Pupil Database. This letter highlighted that "The 
University of Portsmouth team will then provide the names of the children who have not been opted 
out to the Local Authority, who will provide us with the relevant Unique Learner Numbers", which will 
then be used to link to the National Pupil Database. 
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Methodology 


Trial design 
Pupil workshops intervention 


Pupils in Year 5 in six schools were randomly assigned to either the intervention group or the control 
group. Overall 286 pupils were randomly assigned with 144 in the intervention group and 142 in the 
control group. Pupils in the intervention group received a six-week course of mentoring and workshops 
with a focus on growth mindset, from trained university students (PSAs). The PSAs were trained in 
mindset, behaviour management and how to administer the intervention or control workshops, by the 
academic team at the University of Portsmouth. Pupils who were allocated to the control group received 
time-matched general study skills support, also delivered by PSAs. Both intervention and control group 
pupils received a further four-week course of workshops delivered by project partners from the 
Education Business Partnership and Portsmouth Football Club. This ‘active control’ allowed 
identification of the impact of the mindset approach beyond the impact of extra time with mentors and 
external agencies. 


Concerns about contamination were limited since both the treatment and control groups attended 
workshops and were unlikely to be aware that the content of their sessions differed. Hence pupil-level 
assignment was deemed the most appropriate design. It is, of course, feasible that a change in pupil 
behaviour arising from mindset training influenced those in the control group. This was not explored in 
the evaluation, since it would require in-depth and resource-intensive research involving pupils. 


INSET intervention 


Schools involved in the INSET intervention were randomised at the school level. There were 30 schools 
in the INSET intervention with 15 assigned to both the treatment and control group. This meant that 
1,505 Year 5 pupils were involved in the study at the point of randomization (628 in the intervention 
schools and 877 in the control schools). These schools were different from those taking part in the pupil 
intervention. Teachers in the intervention group schools received training which was not available to 
teachers in control group schools. The aim of the intervention was to influence pupil attainment through 
influencing the behaviour of teachers and, by including senior leaders, possibly the whole school 
approach to growth mindset (See ‘Process evaluation’ section), although the impact evaluation looked 
only at outcomes for pupils in Year 5. Learning support staff were also invited to attend the training, and 
a number did so. The nature of the INSET intervention meant that pupil-level random assignment was 
not appropriate, so schools were randomly assigned to receive mindset training or not. 


Eligibility 
Pupil intervention 


Schools for the pupil intervention were drawn from within Portsmouth and were selected to fit the EEF 
eligibility criteria. They had fewer than 60 per cent of pupils achieving Level 4 English and maths at Key 
Stage 2, and lower than average pupil progress. All Year 5 pupils in the selected schools were eligible 
for the trial. 
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INSET intervention 


A larger group of schools was required for the INSET intervention and these were drawn from the 
remaining schools in Portsmouth as well as schools elsewhere in Hampshire and Southampton. A 
workshop was held for interested schools in Portsmouth where it was explained what would be involved 
in the trial. Schools were then asked to sign up to the trial at the end of the workshop. Other schools 
were recruited following telephone calls and in some cases there was an additional school visit. Again, 
all Year 5 pupils in the selected schools were eligible for the trial, via their participating teachers. 


When more schools were required for the trial, recruitment was extended to include schools with a Year 
5 cohort in Hampshire and Southampton Local Education Authorities. The Education Service contacts 
in each area emailed schools to invite them to participate in the trial. Following the initial email, two 
workshops were held for interested schools and they were asked to sign up to the trial at the end of the 
workshop. Other schools were recruited following telephone calls that explained what was involved in 
the trial. 


Intervention 
The Changing Mindsets intervention consisted of two separate interventions. 
Pupil workshops intervention 


This consisted of a six-week course of mentoring and workshops from trained university students, 
working as project support assistants (PSAs). 


Workshops were delivered to both the intervention and control groups and ran over six consecutive 
weeks, each workshop lasting two hours, with the school selecting whether they were to be held in the 
morning or afternoon. The intervention group received mindset workshops, while the control group 
received study skills workshops. Workshops were delivered by three PSAs to groups of 15 pupils at a 
time (a 3/15 staff/pupil ratio). Teachers were not present in the workshops unless a pupil needed 
specialist support, in which case their assigned learning support assistant was present. 


For the first six weeks, the mindset workshops covered: 


Introduction to growth/fixed mindsets 

Types of language (fixed and growth) 

Spelling task—to teach that finding the right strategy helps 
Maths games—to teach that finding the right strategy helps 
Hard work, effort and practice 

Concluding session—consolidation of learning 


OTT Bey OOS 


The study skills workshops covered: 


Exploring different types of learning (e.g. visual, auditory and kinaesthetic learning styles) 
Types of language used about their learning and how to cope with their feelings around these 
Spelling task—developing a new strategy (but no mindset message conveyed) 
Maths—developing a new strategy (but no mindset message conveyed) 

General study skills 

Concluding session—consolidation of learning 


OGL Gok 


The workshops were delivered by 52 paid PSAs. These were undergraduate students (from a range of 
disciplines), many of whom planned to become teachers. Selection criteria included previous 
experience of working with children and all were DBS checked. They were recruited by competitive 
process and trained in mindset, behaviour management and how to administer the intervention or 
control workshops, by the academic team at the University of Portsmouth. It was intended that PSAs 


Education Endowment Foundation 9 


Changing Mindsets 


should be blind to the existence of the two separate workshops—mindsets and study skills. However, 
they were aware that two different programmes were being delivered in the test schools. 


The six weeks of workshops delivered by PSAs were followed by four further weeks of two workshops 
and two visits delivered by project partners from Education Business Partnership? (EBP) and Pompey 
Study Centre? (PSC), now called Portsmouth in the Community (PSC). While pupils in the intervention 
and control groups all took part in sessions delivered by partners and associated visits, these differed 
in that hosts and guides for the study skills control group were not trained in promoting growth mindsets. 
For intervention pupils the hosts and guides were given training in promoting growth mindsets. The 
training helped to ensure that they were therefore able to talk about their challenges, mistakes, growth 
and the 'malleability' of ability using similar terms to those used by PSAs. The aim of these workshops 
was to expose the pupils to successful people living and working in Hampshire who were in careers not 
typically chosen by Portsmouth school-leavers. The aim was to broaden pupil horizons beyond the 
usual, and often limited, choices made by local school-leavers. These four weeks were ordered as 
follows for the pupils in the intervention group: 


Pompey Study Centre in the classroom 

Pompey Study Centre at the football stadium 
Education Business Partnership in the classroom 
Education Business Partnership in the workplace 


PONE 


The ordering of workshops for the control group as follows: 


Education Business Partnership in the classroom 
Education Business Partnership in the workplace 
Pompey Study Centre in the classroom 

Pompey Study Centre at the football stadium 


PwONER 


INSET intervention 


Teachers in the treatment group schools received two teacher training sessions from the project leaders 
at the University of Portsmouth on how to encourage a growth mindset in their pupils. This consisted of 
training teachers in how to teach pupils about the malleability of their intelligence, and how to reinforce 
this in lessons through the way they communicate with pupils. Prior research on growth mindsets has 
suggested that teachers can instil a growth mindset in their pupils by praising them for their effort rather 
than their intelligence. 


The first training session introduced Carol Dweck’s concept of mindset. The trainers covered: the 
theory, the evidence of impact in the US and how to reconsider whole school and classroom culture to 
encourage a growth mindset here in the UK. The changes to classroom culture related to teacher 
expectation, use of language, reward systems and allowing pupils to value learning from mistakes. In 
the second training session (6-8 weeks later) teachers’ experiences of encouraging a growth mindset 
were reviewed, and help was provided to resolve challenges and discuss how they might handle difficult 


2By bringing education and business together, the Portsmouth & South East Hampshire Education Business Partnership Ltd 
(EBP) fosters mutual understanding between the two communities and provides opportunities that help young people to prepare 
for the world of work. By organising business people to present activities in school, and by arranging student visits to business 
and industry, they help to give real life relevance to areas of the curriculum. They also enable students to meet with a wide range 
of people from the world of work and to increase their understanding about the kinds of career opportunities that exist. 

3Using football as a motivational tool, the Pompey Study Centre’s (PSC) core purpose is to provide innovative and challenging 
learning experiences for young people categorised as ‘underachieving’ at Key Stages 2 and 3. Bespoke programmes are 
designed to raise standards in literacy and numeracy through the use of ICT. Learning at the centre takes place during curriculum 
time and also after school. The centre also offers a range of daytime, weekend and holiday programmes including the ‘Pompey 
Double Club’. This highly successful programme is being implemented in local secondary schools and uses football—and 
specifically Pompey, as Portsmouth Football Club is known locally—to encourage reluctant learners. 
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scenarios in order to support the development of a growth mindset. How teachers put this learning into 
practice was at their discretion. The training was designed to give teachers an understanding of the 
general approach and specific techniques, which they could then apply to teaching and learning as they 
thought was appropriate. The INSET intervention was not therefore a manualised approach and was 
similar to the typical teacher professional development session. 


Teachers in the “business-as-usual” control group did not receive an intervention. This does not mean 
that they did not receive any training at all. It means that they did not receive any training in addition to 
what they would normally receive if they were not part of the evaluation. 


Outcomes 


The primary outcome measure was the attainment of year 5 pupils in English and maths, as captured 
by the Progress in English (PiE)* and Measuring Success in Maths (MSiM)° tests. 


Pupils’ theory of intelligence was measured using the three entity theory statements written by Dweck 
(1999).) (see below for details). 


Data for each outcome was collected at three points in time: 


e Baseline test: before the start of the intervention 
e Immediate post-test: four months after completion of the intervention 
e Delayed post-test: ten months after completion of the intervention 


This evaluation report focuses on the post-test conducted ten months after completion of the 
intervention and controls for outcomes prior to the start of the intervention. Results for four months after 
completion outcomes are discussed but not presented. 


The PSAs collected all of the outcome data. While they were also involved in delivering the intervention 
through the pupil workshops, PSAs collected data in different schools from the one in which they 
delivered the intervention. They were therefore not aware of whether pupils had received mindsets or 
the study skills (control) intervention. No PSAs were involved in the INSETs, so that data collection was 
carried out blind with regard to whether schools were in the intervention or control group. Scoring of 
tests was initially conducted by Research Fellows at the University of Portsmouth, who then trained 
University of Portsmouth research assistants in the scoring approach. Research assistants then carried 
out most of the scoring. 


4hittp://www.gl-assessment.co.uk/products/progress-english 


7 http://www.gl-assessment.co.uk/products/measuring-success-maths 
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Progress in English (PiE) 


Pupils’ reading and writing ability were measured at baseline and immediate post-test using Progress 
in English 9 (Key Stage 3 and 4). At the delayed post-test Progress in English 10 (Key Stage 4) was 
used. 


PiE 9 and 10 both comprise four exercises, aimed to examine pupils’ spelling, grammar and reading 
comprehension (both narrative and non-narrative). At all three testing points, Exercise 3 (narrative 
reading comprehension) was dropped so the maximum score that pupils could obtain was 28. This 
decision was taken after discussion with representatives at GL Assessments and based on the time 
constraints of the testing sessions. We were told that it was possible to remove sections of the test and 
still obtain a standardised test, but we later found out that this was not the case. Therefore, pupils’ 
results needed to be standardised within the current sample at each time point. 


Pupils were guided through the first section—a spelling test—with the passage being read aloud to 
pupils twice for them to fill in the blanks. The second exercise was self-guided with pupils required to 
correct grammatical errors in a passage of text. The final section examined pupils’ reading 
comprehension, requiring them to read a passage independently and then answer eight multiple choice 
questions. 


Measuring Success in Maths (MSiM) 


At all three time points, pupils’ numeracy was assessed using the MSiM written test for Year 5 children. 
This test was selected as it is intended for monitoring progress throughout the school year and is 
informed by past national tests. The Year 5 test has been designed to assess National Curriculum Level 
3 to Level 5 maths, covering areas such as: counting, calculating, shapes, measurements, handling 
data and understanding number facts. The test is scored out of a total of 75. 


Mindsets 


Dweck’s original questions include six items comprising three entity theory questions and three 
incremental theory questions. However, owing to the age of the sample and longitudinal nature of the 
study, Dweck (1999) recommends using the entity-only scale, as the incremental questions are more 
likely to suffer from social desirability, especially when repeated over time. The entity theory question 
statements are as follows: 


e You have a certain amount of intelligence and you really can’t do much to change it. 
e Your intelligence is something about you that you can’t change very much. 
e You can learn new things, but you can’t really change your basic intelligence. 


For each of these statements, pupils were read the statements and asked to rate the degree to which 
they agreed with each statement using a 6-point Likert scale, ranging from 1 (strongly disagree) to 6 
(strongly agree). Ratings for each question can then be totalled whereby lower scores represent the 
belief that intelligence can be changed and higher scores indicate a more ‘fixed’ view of intelligence. 


Sample size 


The aim of the pupil intervention trial was to recruit six schools with pupils in each school randomised 
into treatment and control groups. The power calculations assumed 40 pupils per school, 0.05 
significance level, 0.8 power and an intra-cluster correlation of between 0.1 and 0.15. The minimum 
detectable effect size was estimated to be in the range of 0.4 to 0.5 standard deviations. 


For the INSET intervention trial the aim of the project was to recruit 30 schools to the study with 15 
schools randomised into treatment and control groups. The power calculations assumed 40 pupils per 
school, 0.05 significance level, 0.8 power and 0.1 to 0.15 intra-cluster correlation. The minimum 
detectable effect size was estimated to be in the range of 0.3 to 0.4 standard deviations. 
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These calculations did not factor in stratification. 


Randomisation 
NIESR randomised the pupils and the schools. 


For the pupil intervention only first names for eligible pupils were available at the time of randomisation, 
so no stratification was possible. Randomisation of pupils (to achieve a 50:50 allocation) was performed 
as follows. 


In each school, the names of eligible pupils were sorted alphabetically by the first name of the pupil. 
The pupil whose first name was first in the alphabet was allocated number 1 with the pupil whose first 
name was next in the alphabet allocated number 2, through to the pupil whose first name was last in 
the alphabet being allocated number N (the number of eligible pupils in each school). 


e Each pupil was then assigned a randomly generated number. 

e Pupils in each school were then sorted by the random number. 

e The first pupil was randomised into either the treatment or control group. 

e Each subsequent pupil was assigned to have the opposite outcome of the previous pupil. 


For the INSET intervention randomisation was carried out within blocks defined by area (Southampton 
8 schools, Portsmouth 12 schools or Hampshire 10 schools) and by the proportion of pupils in each 
school shown to have achieved Level 4 or higher at Key Stage 2 in both English and maths in the 2011 
school performance tables. Attainment data was not available for all schools in Southampton, so all 
schools in Southampton formed one block. Schools in Portsmouth and Hampshire were each split into 
two blocks based on low or high attainment. In Portsmouth high attainment schools were those where 
more than 74% of pupils achieved Level 4 in their Key Stage 2 assessment in 2010/11; the equivalent 
figure in Hampshire was 65% (the thresholds were chosen to achieve equal sized groups in each area). 
This resulted in five blocks (or strata). 


Randomisation of schools (to achieve a 50:50 allocation) was performed as follows: 


e Each school was assigned a randomly generated number. 

e Schools were sorted by blocking variable and, within each block, by the random number. 

e The first school was randomised into either the treatment or control group. 

e Each subsequent school was assigned to have the opposite outcome of the previous school. 


Analysis 


For both interventions the outcome variable was the standardised test score ten months after the 
intervention. 


For the pupil workshop intervention the analysis used regression models to compare pupil outcomes in 
the intervention and control groups. Baseline covariates were also included to capture pupils’ age in 
months, age in months squared, gender, eligibility for FSM, pre-test score and the share of pupils in the 
school with Special Educational Needs (SEN) and the share of pupils in the school with English as an 
Additional Language (EAL). 


For the INSET intervention the analysis used multilevel regression models to reflect the clustered nature 


of randomisation. The same baseline covariates as described above were included in the models as 
well as dummy variables identifying the blocks of schools used in the randomisation of the schools. 
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For both interventions separate analysis was conducted for pupils eligible for FSM and pupils in the 
bottom third of the distribution of pre-test scores. Separate models were estimated for these sub- 
samples. Here the reduced sample size reduces the experimental power in the models. 


The impact was estimated following the intent-to-treat principle. However, schools that dropped out of 
both trials did not collect data on outcomes so this was not possible. Estimation results are considered 
in terms of effect sizes calculated by dividing the estimated impact coefficients by the level 1 standard 
deviation from the respective multi-level regression, and so control for covariates and the school-level 
random effect. 


Process evaluation methodology 


NIESR conducted the qualitative evaluation of Changing Mindsets which included a process evaluation 
and qualitative research on the programme’s impact and effectiveness. This was with a view to 
identifying features contributing to successful implementation and to understanding participants’ 
experiences of the intervention. 


In the context of the project, participants included PSAs, teachers and business partners from 
Portsmouth FC and Portsmouth Education Business Partnership. Representatives from each of these 
groups were interviewed. The perspectives of pupils were not included, although classroom observation 
was carried out to assess pupil engagement, understanding of mindset and study skills concepts and 
their application to the various exercises which formed part of the sessions. Qualitative research also 
aimed to bring greater clarity to the quantitative research findings and to understanding the reasons 
behind any impact, or absence of proven impact. NIESR also used data collected by Portsmouth 
University on teachers’ views on the training, which included whether they had previously had training, 
or had practised mindset or similar approaches. This included closed and open-ended questions. 


The qualitative evaluation included the following elements: 
For the pupil workshops intervention: 


e Interviews with 12 project support assistants and Research Fellows in January 2013 

e Observation of 2 pupil workshops in March 2013 

e Interviews with 4 headteachers and Year 5 class teachers in 5 selected pupil-intervention 
schools in June/July 2013 

e Interviews with 3 project partners from Portsmouth FC and Portsmouth Education Business 
Partnership in July 2013 


For the INSET intervention: 


e Observation of INSET sessions in 2 schools in April 2013 

e Interviews with 8 teachers in 6 selected INSET intervention schools in July 2013 

e Survey of INSET schools in March 2014 (responses from 9 out of 12 schools) 

e Evidence on the practices of control schools, in particular awareness of mindset (collected by 
Portsmouth University in May 2014) 


All activities were carried out by NIESR, with the exception of evidence from control schools, collected 
when schools received mindset training. Portsmouth University also collected training evaluation data 
which was forwarded to NIESR and incorporated into the external evaluation. As stated above, this 
included teachers’ responses to closed questions on aspects of the training and responses to open- 
ended questions about their knowledge of growth mindset approaches. 


Data collected for the process evaluation was largely qualitative. Therefore qualitative data analysis 
methods were used. These are described in the ‘Process evaluation’ section of the report. 
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Impact evaluation 


Timeline 


Schools were recruited up to the end of October 2012, with randomisation of pupils in the pupil 
intervention and schools in the INSET intervention both carried out in November 2012. 


Baseline assessments in all schools were conducted between the week commencing 7 January 2013 
and week commencing 28 January 2013. 


The pupil intervention took place between February and May 2013. 


For the INSET intervention the first half-day training took place in January/February 2013, after the 
completion of baseline assessments for pupils in these schools, with the second half-day training in 
April 2013. 


The initial post-test was administered in June and July 2013 and the final post assessment was 
administered between February and April 2014. All assessments were conducted by PSAs employed 
by the University of Portsmouth. 


Participants 
Recruitment 


The six schools in the pupil intervention were all involved in the initial bid for funding of the intervention. 
The EEF asked for the pupil and teacher interventions to be split so more schools were required. 
Portsmouth City Council invited 128 schools from around Hampshire to participate in the intervention. 


Initially a workshop was held for interested schools in Portsmouth where it was explained what would 
be involved in the trial. Schools were then asked to sign up to the trial at the end of the workshop. Other 
schools were recruited following telephone calls and in some cases there was an additional school visit. 


When more schools were required for the trial, recruitment was extended to include schools with a Year 
5 cohort in Hampshire and Southampton Local Education Authorities. The Education Service contacts 
in each area emailed schools to invite them to participate in the trial and the recruitment process was 
then the same as in Portsmouth. Following the initial email, two workshops were held for interested 
schools and they were asked to sign up to the trial at the end of the workshop. Other schools were 
recruited following telephone calls that explained what was involved in the trial. 


In total 30 schools were recruited to the INSET intervention and six schools recruited to the pupil 
intervention. The geographical distribution of the schools involved in the INSET intervention is 
presented in Table 2. All six schools involved in the pupil intervention were located in the Portsmouth 
area. 


All pupils in the Year 5 cohort in these schools were eligible to participate in mindset, study skills or, in 
the case of the INSET intervention, via their class teacher. 


Numbers in the trial 


Figures 1a and 1b summarise the number of schools and pupils involved in the trial for the pupil and 
INSET intervention respectively. 


For the pupil trial (Figure 1a) randomisation was carried out for 286 pupils in six schools, with 144 
randomly allocated to the intervention group, and 142 randomly allocated to the control group. One 
school dropped out because it felt that the PSAs did not manage behaviour effectively, meaning that 
28 pupils (intervention group) and 29 pupils (control group) dropped out. For the intervention group a 
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further 9 pupils moved schools, with 3 pupils moving school in the control group. The overall dropout 
rates were 26% and 23% for the intervention and control groups respectively. 


This left five schools, with 107 and 110 pupils allocated to the intervention and control group 
respectively. For both groups 16 pupils were unavailable for testing. 


Similarly for the INSET trial (Figure 1b) 128 schools were invited to participate, and 30 schools agreed 
to participate. Of these 15 were randomly allocated to the intervention group and 15 to the control 
group. Of the 15 intervention schools, 2 schools dropped out after the programme started. One of 
these was having a difficult time following poor inspection reports and despite receiving the teacher 
training dropped out before the final assessments. In the other school, University of Portsmouth staff 
turned up for training on two occasions to discover something else had been arranged so were sent 
away. Of the 15 control schools 4 dropped out, also after the intervention had started. One of these 
was due to school restructuring so they were too busy to continue with the study; a second could not 
find time to conduct either of the post-intervention assessments; and a third did not like the 
assessments and decided that the trial was of no value to the school. No reason was given by the 
fourth drop-out school. 


This left 13 and 11 schools involved in the intervention and control groups, corresponding to 87% and 
73% of those randomised, and with 476 and 658 pupils respectively. 


The impact estimates were based on fewer observations due to pupils moving schools and missing 
assessment data; in all but two cases this was due to pupil absence on assessment days, but for two 
pupils parental consent to undertake the assessments was denied. 
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Figure 1a: Flowchart of sampling, allocation and attrition, Pupil workshops 
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Figure 1b: Flowchart of sampling, allocation and attrition, INSET 
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School and pupil characteristics 


Tables 1, 2a and 2b present the characteristics of the schools included in the trials. 


Table 1 shows the geographical distribution of schools in the INSET intervention. By design this was 
balanced across the intervention and control groups. 


hical distribution of schools in INSET 
Intervention Control | 
Hampshire 5 33.3 5 ; 


Total 15 100 15 100 


Table 1: Geograp 


Table 2a presents characteristics for the schools recruited for the pupil intervention alongside data for 
England. All six schools involved in the trials were located in the Portsmouth area. The data shows that 
intervention schools compared to all schools in England had a much higher percentage of pupils eligible 
for free school meals (42.2% compared with 19.8%) and a lower percentage of pupils who achieved 
Level 4 or higher in reading, writing and maths (55.7% compared with 73.5%). 


Table 2a: Characteristics of pupils in recruited schools il interventions 
England Difference 


Talccyavd-yayicey a] 
rXoq aLele) (3) 
(Portsmouth) 
Number of schools N=6 N=15,440 
(68.1) (145.6) [59.4] 


Percentage of pupils with SEN statement 8.8 12.7 -3.9 


(3.0) (19.5) [8.0] 
Percentage of pupils with EAL 16.9 14.9 2.0 

(11.3) (22.6) [9.9] 

(15.3) (15.6) [6.4] 


Percentage achievement Level 4 or above in reading, 55.7 
writing and maths 


(11.7) (19.0) [7.7] 


Notes: Based on School Performance Table 2011/12, Department of Education. Standard Deviation in (.). 


The average size of schools in the pupil intervention was 259 pupils, slightly bigger than the average 
school size in England. The intervention schools had a lower percentage of pupils with a SEN statement 
than all schools in England (8.8% compared with 12.7%), but had a slightly higher percentage of pupils 
with English not as their first language (16.9% compared with 14.9%). 


Table 2b shows the mean characteristics of the schools recruited for the INSET intervention. The 
intervention and control schools look similar. The average number of pupils in the intervention group 
schools was 239, slightly lower than in the control group schools (304). In both intervention and control 
group schools, about 10% of pupils were under the Special Education Need statement or on School 
Action Plus (SEN). In the intervention group schools about 21% of pupils did not have English as their 
first language (EAL), while in control group schools the percentage was lower at 12%. The percentage 
of pupils eligible for free school meals (FSM) was 30% and 26% in the intervention and control group 
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schools respectively, and just over 70% of pupils in both groups achieved Level 4 or higher in reading, 
writing and maths. 


ils in recruited schools (INSET 
Intervention Control Difference 
Number of schools 15 15 
Size of schools (number of pupils) 239 304 -65 
eae (105) [34] 
Percentage of pupils with SEN statement 10.7 -1.2 
a (5.2) [1.7] 
(14.3) (16.7) [6.1] 
(10.3) (15.8) [4.9] 


Percentage achievement Level 4 or above in reading, 74.5 
writing and maths? 


(10.9) (12.7) [4.4] 
Ofsted rating? 2.43 2.36 0.07 
(0.76) (0.74) [0.28] 


Notes: Based on School Performance Table 2011/12. Department of Education.Standard Deviation in (.).Standard Errors in [.]. 
1. There were three intervention group schools where data on the percentage of pupils with EAL was suppressed. 

2. For one of the control group schools there was no data on the percentage of pupils achieving Level 4 or above in reading, 
writing and maths in 2011/12. 

3. For one intervention group and one treatment group school there is no Ofsted inspection data 


Table 2b: Characteristics of pu 


Baseline pupil characteristics in the academic year 2012/2013 based on the National Pupil Database 
(NPD) for the pupil and INSET interventions are presented in Tables 3a and 3b respectively. 


Table 3a shows baseline characteristics for the pupil trial. Here 46% of the intervention group and 43% 
of the control group were female and the mean age of pupils in each group was 9 years old at the start 
of the academic year 2012/13. Roughly 40% of the pupils in the intervention group were eligible for 
FSM compared with 34% of the control group. 


Table 3a: Baseline pupil-level characteristics, treatment and control groups (pupil) 
Intervention Control Difference 
(0.501) (0.498) [0.072] 
(0.278) (0.339) [0.044] 
(0.492) (0.477) [0.069] 


For the INSET trial (Table 3b), roughly half the pupils in both the intervention and control groups were 
female and the average age of both groups was 9 years old at the start of the academic year 2012/13. 
In the intervention schools around 29% of pupils were eligible for FSM; this percentage is lower (15%) 
in the control schools. 


Table 3b: Baseline pupil-level characteristics, treatment and control groups (INSET) 
Intervention Control Difference 


(0.501) (0.500) [0.032] 


Mean age of pupils (years) 9.096 9.100 -0.004 
(0.295) (0.300) [0.019] 

Proportion of pupils eligible for FSM 0.291 0.156 0.135 
(0.455) (0.363) — [0.025] 


Notes: Based on the National Pupil Data. Department of Education. Standard Deviation in (.).Standard Errors in [.]. 
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Tables 4a and 4b report the baseline levels of the primary outcome variables, and the mindset scores. 
Scores are similar for the intervention and control groups in the pupil intervention. 


Table 4a: Pre-test for English and maths, treatment and control groups (pupil 
Intervention Control Difference 
English 15.453 14.1 1.305 
47 
(6.614) (7.6 [1.041] 
94 


Number of pupils 95 95 190 
Maths 12.872 13.2  -0.426 
98 
(8.800) (10. = [1.425] 
655) 


Number of pupils 94 94 188 
Mindset 11.233 12.1 -0.893 
26 
(3.672) (3.7 [0.544] 
22) 


Number of pupils 90 95 185 


Notes: Standard Deviation in (.). Standard Errors in [.]. 


Comparison of maths and English in the intervention and control schools in the INSET trial (Table 4b) 
shows that pupils in control schools perform better in both English and maths and were less likely to 
agree with the fixed mindset questions. 


Table 4b: Pre-test for English and maths, treatment and control groups (INSE 
Intervention Control Difference 


English 16.010 16.847 -0.837 
(6.757) (7.077) [0.451] 
Number of pupils 402 582 984 


Maths 10.945 13.783 = -2.837 


(8.202) (9.515) — [0.585] 
Number of pupils 403 575 978 
(3.435) (3.492) [0.227] 


Number of pupils 396 566 962 
Notes: Standard Deviation in (.). Standard Errors in [.]. 


Outcomes and analysis 
The main results are presented in Tables 5a —5d. 
Pupil intervention 


Effect sizes are presented in Table 5a for the pupil intervention, full estimation results are shown in 
Appendix Tables Ala—Aic. Pupils with missing assessment data have not been included in the 
analysis. No significant effect on maths or English is evident. This is also true for the FSM and low pre- 
test subgroups. Here, the effect size for the low pre-test subgroup for maths and English was however 
considerably larger than the effect size for the full sample. Table 5b reports similar results for mindsets. 
Here a negative effect size means a change to more of a growth mindset. These results suggest no 
significant impact on mindsets. 
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Table 5a: Estimation results for English and maths (pupil intervention) 
Raw means Effect size 


Intervention group Control group 


Outcome n Mean n Mean n in model Hedges g** p- 
(missing) (95% Cl)* (missing) (95% Cl)* (intervention; (95% Cl) value 
control) 

Maths 90 (6) 20.0 89 (10) 20.2 174 (88; 86) 0.10 (- 0.24 
(17.5, (17.0, 0.07, 0.27) 
22.4) 23.3) 

English 90 (6) 15.4 91 (8) 13.6 178 (89; 89) 0.18 (- 0.07 
(14.1, (12.2, 0.02, 0.37) 
16.8) 15.1) 


FSM subgroup 


Maths 34 (4) 17.5 34 (6) 16.4 61 (33; 28) 0.11 (- 0.39 
(13.1, (10.9, 0.14, 0.36) 
21.9 21.8 

English 38 (4) 13.1 34 (4) 11.5 (9.3, 64 (34; 30) 0.17 (- 0.33 
(10.5, 13.7) 0.18, 0.52) 
15.6) 

subgroup 

25 10.6 (8.3, 30 (5) 8.9 (6.8, 55 (25; 30) 0.38 (- 0.14 

12.9) 11.0) 0.13, 0.88) 

English 25 9.8 (7.4, 31 (4) 7.8 (6.0, 57 (24; 33) 0.34 (- 0.17 
12.1) 9.6) 0.15, 0.84) 
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Table 5b: Estimation results for mindset (pupil intervention) 
Raw means Effect size 


Intervention group Control group 


Outcome n Mean n Mean nin model Hedges p- 
(missing) (95% (missing) (95% (intervention; g** value 
Cl)* Cl)* control) (95% Cl) 
Full sample 89 (7) 10.0 91 (8) 11.6 172 (83; 89) -0.29 (- 0.05 
(9.1, (10.8, 0.58, 


10.9) 12.4) 0.00) 

FSM 34 (4) 10.8 11.9 62 (32; 30) -0.13 (- 0.60 
subgroup (9.7, (10.5, 0.64, 
12.0) 13.3) 0.37) 

Low pre-test 12.9 52 (21; 31) -0.30 (- 0.38 
subgroup ; (11.6, 0.97, 
14.1) 0.38) 


Inset intervention 


Table 5c presents impacts on the primary outcomes as effect sizes for the INSET intervention; full 
estimation results are shown in Appendix Tables A1ld—A1f. The results suggest that despite significant 
differences between the intervention and control groups in raw scores, once control variables are 
included there was no significant effect on maths or English. This is also true for the FSM and low pre- 
test subgroups. The effect size for the low pre-test subgroup for maths was however considerably larger 
than the effect size for the full sample but not significant. 


Table 5c: Estimation results for English and maths (INSET intervention 
Raw means Effect size 


Intervention group Control group 


Outcome n Mean n Mean nin model Hedges g** p- 
(missing) (95% (missing) (95% (intervention; (95% Cl) value 
Cl)* Cl)* control) 
Maths 376 (50) 20.5 568 (42) 24.2 896 (358; 538) 0.01 (- 0.90 
(19.2, (23.0, 0.18, 0.21) 
21.7) 25.3) 
English 379 (47) 16.0 550 (60) 17.2 885 (359, 526) -0.11 (- 0.21 
(15.3, (16.7, 0.28, 0.06) 
16.6) 17.8) 
FSM subgroup 


Maths 109 (15) 18.2 83 (12) 20.0 178 (102; 76) 0.04 (- 0.76 
16. (16.8, 0.23, 0.31) 
23.2) 
English 110 (14) 14.3 80 (15) 15.0 176 (103, 73) -0.01(- 0.96 
(13.0, (13.4, 0.24, 0.23) 
15.6) 16.6) 


Maths —119 (16) 12.6 256 (119;137) 0.22 (- 0.22 
(11.4, 0.13, 0.57) 
13.9) 
English 119(16) 104(9.6, 134(14) 11.0 292(121,171) -0.03(- 0.89 
11.2) (10.3, 0.41, 0.35) 
11.7) 
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The effect on mindset was also not statistically significant for the full sample or the low pre-test 
subgroup, but was significant for the FSM subgroup. Here a fall in the mindset score indicates a shift 
towards more of a growth mindset for pupils who were eligible for free school meals. 


Table 5d: Estimation results for mindset (INSET intervention) 
Raw means Effect size 


Intervention group Control group 


Outcome n Mean n Mean nin model Hedges p- 
(missing) (95% (missing) (95% (intervention; g** value 
Cl)* Cl)* control) (95% Cl) 
372 (54) 9.4 567 (43) 9.4 878 (350; 528) 


Full sample 


(8.9, 
9.8) 


FSM 110 (14) 
subgroup 


Low pre-test 119 (16) 


subgroup 


80 (15) 10.9 


134 (14) 11.1 


(9.1, 
9.7) 
176 (102; 74) 
(10.0, 
11.8) 
225 (105; 120) 
(10.5, 


11.7) 


Cost 


Schools involved in this evaluation received funding from the EEF to participate in the programme. The 
cost estimates presented here relate to the cost to schools if they had covered the costs of the 
intervention themselves, without receiving any funding from the EEF. 


The cost information used to calculate these estimates was provided by the University of Portsmouth 
team. 


Pupil workshops intervention 


The cost data provided by the University of Portsmouth included the cost of delivering the interventions 
to both the control and intervention group. The workshops delivered in the intervention group were 
identical to those delivered to the control group, apart from the focus on growth mindsets. This means 
that a cost per pupil can be estimated by dividing the total cost of delivery of all the workshops by the 
number of pupils in both the intervention and control groups (286). This gives a cost per pupil estimate 
of £397. 


Cost item Cost of delivery to 


Talcsiavcciaviceyam-laremee)alige)| 
group 


Cost per pupil 


University of Portsmouth staff £48,057 


Portsmouth Football Club and 
Portsmouth Education Business 
Partnership four weeks’ 
intervention delivery 


£37,000 


Travel and subsistence £3,098 


Undergraduate workshop tutors 


£23,358 
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Consumables £1,000 £3 
4 


Technical support £1,005 £ 


Total £113,519 £397 


Teacher training intervention 


The control group in this case received business-as-usual treatment, so the cost per pupil estimate is 
the total cost of delivering the teacher training divided by the number of pupils in the intervention group 
(628). The cost per school estimate was calculated by dividing the total cost of delivering the teacher 
training by the number of schools in the intervention group (15). 


Cost item Cost Cost per pupil Cost per school 
University of Portsmouth staff £29,314 £47 £1,954 
time 


Consumables £1,000 £2 £67 
£14 


Technical support £209 £0.33 


Total cost £30,524 £49 £2,035 


Teachers required two half-days of supply cover to participate in the teacher training. The cost of 
providing supply cover was excluded from this estimate, as schools arrange cover in various ways that 
may or may not incur a financial cost. 


The teacher training intervention is an investment in the skills of a school’s teaching staff that will last 
several years. Teachers in England complete an average of six years at a single school before they 
leave (Allen et al., 2012). This means that teachers will stay at their current school for an average of 
three more years after they have completed the growth mindsets training. Three years therefore seems 
a Suitable estimate of the average amount of time before schools will need to pay for the training to be 
refreshed. The cost of the teacher training intervention over three years is £16 per pupil per year. 


Cumulative Cost per pupil per Cumulative cost Cost per school 
cost per pupil WEE per school 


49 49 2035 2035 


97 24 4070 1017 
16 


146 6105 678 
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Calculating delivery costs 


Some of the costs provided by the University of Portsmouth were for cost items that were used in both 
the teacher training and the pupil workshop interventions. As the University of Portsmouth team were 
also involved, some of the costs applied to delivery staff who were involved in both delivery of the project 
and the provision of testing, but the costs provided for these staff did not distinguish between the two 
aspects. This meant it was challenging to isolate the cost of staff time spent in delivering the project. 


The University of Portsmouth team estimated the amount of time spent by their staff on different aspects 
of the project. 


Pupil Pupil INSET 
intervention | intervention | intervention | INSET 
delivery assessment | delivery evaluation | Total 
% 
project 
hours 45 8 9 38 100 


The estimates of the time spent on different parts of the project were used to allocate the staff costs to 
the different parts of the project. 


Future cost of the programme 


The University of Portsmouth has developed a programme that combines the two interventions tested 
here and can be delivered in all key stages. This programme provides: 


e a full day’s training for teachers 

e the manual, lesson plans and classroom materials required for schools to run the six-week pupil 
workshop intervention themselves 

e six months’ access to advice and support service for the school. 


The charge for this is £1,800 per school, plus travel and accommodation costs for two trainers if is 
school is located more than 20 miles outside of Portsmouth. No additional resources are required to 
deliver the intervention, but additional mindset story books, lesson plans and materials and data 
analysis services are available at additional cost. 
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Process evaluation 


The purpose of the process evaluation was to identify features contributing to successful 
implementation and to understand participants’ experiences of the intervention. Qualitative research 
within the process evaluation also aimed to bring greater clarity to the quantitative research findings 
and to understand the reasons behind any impact, or absence of proven impact. Our methods included 
the following: 


For the pupil intervention: 


e Interviews with 12 PSAs and Research Fellows in January 2013 

e Observation of 2 pupil workshops in March 2013 

e Interviews with 4 headteachers and Year 5 class teachers in 5 selected pupil intervention 
schools in June/July 2013 

e Interviews with 3 project partners from Portsmouth FC and Portsmouth Education Business 
Partnership in July 2013 


For the INSET intervention: 


e Observation of INSET sessions in 2 schools in April 2013 

e Interviews with 8 teachers in 6 selected INSET intervention schools in July 2013 

e Survey of INSET schools in March 2014 (responses from 9 out of 12 schools) 

e Evidence on the practices of control schools, in particular awareness of Mindset (collected by 
Portsmouth University in May 2014) 


All activities were carried out by NIESR, with the exception of evidence from control schools, collected 
when schools received Mindset training. Portsmouth University also collected training evaluation data 
which was forwarded to NIESR and incorporated into the external evaluation. This included teachers’ 
responses to closed questions on aspects of the training and responses to open-ended questions about 
their knowledge of growth mindset approaches. 


Data collected through the range of approaches described above was analysed using a framework 
approach. This enables the analysis of qualitative data in a written form, and is therefore appropriate 
for the analysis of transcripts of interviews with teachers and other project participants, as well as 
research notes taken during observation of INSET and classroom interventions. Qualitative responses 
to survey questions were also analysed in this way. The method entails coding the data into themes 
and issues. In this case, codes were a mixture of predetermined ones, developed during the design of 
the process evaluation and taking account of the aims of the intervention, as well as those that emerged 
from the text of transcripts and observations. Codes identified different types of information, for example 
more tangible ones such as knowledge of mindset, experiences of the training and of putting the 
approach into practice as well others such as values and feelings. Throughout the analysis process, we 
looked for similarities and differences in the data. The framework approach allows for tracts of text to 
be classified under more than one code, and codes were, in some cases, amalgamated to form wider 
groups, particularly where substantial issues were concerned. 


The codes and groups developed in the analysis of data formed the analytical framework and were 
used to structure the findings into a preliminary report. We then restructured this document to follow the 
format required by the EEF. This involved structuring the findings using the main EEF process 
evaluation criteria: implementation, outcomes and fidelity. 
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Implementation 
Implementation of the pupil intervention 


As described earlier, the pupil intervention involved project support assistants (PSAs) delivering either 
intervention (mindset) or control (study skills) sessions to pupils in test schools over a six-week period, 
following training in either approach. PSAs also delivered the tests. 


From the PSAs' perspectives, the intervention had generally gone smoothly: they felt well prepared, 
supported by the project team at Portsmouth University, and they delivered the sessions as planned. 
Feedback from schools involved in the pupil intervention was largely positive. Schools had been keen 
to take part in the project, believing that it could benefit pupils. Organisational and planning aspects 
were reported to work well, with the only logistical difficulties reported by schools with mixed Year 5 and 
6 groups and who found random allocation difficult for organisational reasons. 


Observations of Changing Mindsets by the NIESR team in one school found both sessions to be 
delivered effectively by the PSAs. Activities were well structured and explained. Behaviour in both 
groups was good, although some pupils in the Changing Mindsets group did require active 
management. The ability of PSAs to manage the behaviour of some pupils was an issue of concern 
among some of the pupil intervention schools. Teachers suggested that PSAs might have benefited 
from the opportunity to visit the school before the project and observe lessons. 


Barriers and enablers to implementing the pupil intervention 


The pupil intervention was implemented as planned with no significant barriers. The use of PSAs 
combined both testing and delivery in the intervention schools in a relatively low-cost and consistent 
way. Schools appeared to cope with the logistical demands of the project. Some teachers in the pupil 
intervention schools expressed concern about pupil management, as described above, but this did not 
impede delivery. No problems were reported with the sessions delivered by partners from Portsmouth 
FC and the Education Business Partnership. These partners were pleased to have been involved in the 
intervention. 


There is no evidence to suggest that PSAs were more or less effective in delivering mindsets or study 
skills approaches to pupils than, for example, class teachers. The main limitation to using PSAs to 
deliver the programme is the lack of follow-through within teaching and learning. As we explain below, 
the relatively short duration and limited intensity of the programme is likely to explain the absence of a 
measurable impact. This includes the absence of a longer, sustained and manualised approach in the 
pupil intervention, rather than the use of PSAs as such. 


While not strictly relevant to the outcomes of the project, it is important to acknowledge that the 
involvement of PSAs had wider benefits. In particular it helped students acquire classroom experience 
for applying to teacher training. Less concretely and possibly of more relevance to the question of 
impact, some PSAs commented that their connection with Portsmouth University had been of interest 
to some pupils. It might therefore be speculated that there could be additional, unmeasured and longer- 
term benefits in exposing children to external visitors from a local university. This might be all the more 
effective given the association which they may have formed between contact with the university, Growth 
Mindsets and aspirations. 


Implementation of the INSET intervention 


INSET was delivered to teachers in two separate sessions in the test schools. The project team at 
Portsmouth University asked schools to require the attendance of Year 5 teachers, school leaders and 
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learning support staff but that other staff would also be welcome. This resulted in attendance of a variety 
of staff, including teachers and learning support staff across years as well as leaders. 


Observations by NIESR researchers of INSET sessions in two schools found the training to be clear 
and engaging in content and delivery. Teachers in both sessions supported the theory of growth 
mindsets and were committed to using the approaches: both classroom teachers and leaders were 
interested in creating a ‘praise environment’. Teachers expressed some concern that the schools’ 
efforts could be undermined by parents’ continuing practice of praising the child, not the effort, and 
discussed ways in which parents might become involved. 


The INSET training was intended not to be overly prescriptive by telling participants how to follow the 
Changing Mindsets approach in any detail, but to adopt three or four approaches. This was described 
by the Portsmouth team as 'making a difference without overburdening' the teachers and other staff 
involved. 


Many of those who attended the INSET training said they were already familiar with the work of Carol 
Dweck in relation to Growth Mindsets, or similar work of Guy Claxton (Building Learning Power) or 
Shirley Clarke (Formative Assessment). However, they had not implemented these approaches, 
particularly those of Carol Dweck, in the classroom. Therefore, INSET was clearly essential in equipping 
participants with the knowledge and skills to do this. 


Attendees found the INSET interesting, engaging and relevant to their teaching. They felt it delivered 
an appropriate mix of theory and practical ideas for delivering the approach. In particular, participants 
commented that it gave them the terminology, and particularly the praise vocabulary. As well as learning 
the theory and language of mindset, teachers particularly valued gaining ideas for teaching and 
learning, and for wider use in display boards, assemblies and whole school work. Participants felt the 
school as a whole would benefit from the INSET they had received and that it was a worthwhile use of 
their own time. Some teachers felt encouraged to find out more about the approach through further 
reading, following up references provided during the INSET. 


Through developing knowledge and skills associated with the mindset approach, the INSET created 
the necessary conditions for the success of the project in INSET schools and undoubtedly made it 
attractive to teachers, learning assistants and school leaders. Feedback from the first session indicated 
that teachers wanted more practical ideas and examples of its application, but at the same time had 
gained many ideas which they planned to implement in the classroom. Most participants in the first 
INSET said they intended to use the resources provided. By the time of the second INSET three months 
later participants had put some mindset features into practice and valued the opportunity to share 
experiences and strategies and gain further ideas for activities and use of resources. 


Barriers and enablers to adopting Mindsets 


Few barriers were identified to the implementation of mindset in the INSET schools in this stage of the 
evaluation. Some teachers in lower year groups felt that the approach and materials would need to be 
adapted for younger age groups. Barriers to its effectiveness were identified in lack of reinforcement or 
undermining of mindset principles by parents. 


Teachers in INSET schools described the approach as easy to implement, with the approach fitting very 
well with the ethos of the school and with the potential to address the key barriers to learning. These 
were principally social deprivation, low expectations, lack of self-belief and difficulty in identifying with 
academic success. The closeness of this fit had led schools to encourage the approach across all year 
groups, through adoption of similar language around learning, among both teaching and support staff. 


Although most teachers interviewed said they had not heard of the Mindsets approach as such before 
the training, they also said that they were doing something similar, meaning that they praised pupils’ 
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efforts and promoted resilience. However, they also said their approaches had lacked the specific 
growth mindsets vocabulary. The INSET training also appears to have encouraged more frequent and 
consistent use of the approach and assisted the process of implementation. 


All teachers interviewed in the INSET schools felt that Mindsets is well suited to their school and pupils, 
which is likely to have helped to ensure it was delivered as intended. They also intended to continue to 
use it, which again suggests that schools were committed to delivery. However, longer-term use was 
seen to depend on factors such as the demands of other classroom-based initiatives. A further barrier 
was identified in staff turnover and the need to continually reinforce the approach among staff, an issue 
which might be addressed through embedding within a whole school approach, as one respondent 
suggested. 


These views were reinforced by survey responses from 9 out of the 12 INSET schools in March 2014. 
The survey showed that all of these schools had been using Changing Mindsets since they received 
INSET training the previous year. In the year 2012-13 all schools had used it in Year 5, and four had 
also used it in Years 4 and 6. The following year they had also used it in all school years, but more 
with older year groups. The evidence base of Mindsets was seen as important to the decision about 
whether or not to use it. Teachers had largely gained this evidence through participation in the project, 
particularly the INSET training, and generally found it convincing. 


In trying to understand the absence of a quantitative impact, it should be asked whether teachers’ and 
schools’ previous use of similar approaches may have weakened the impact of the INSET intervention, 
and possibly the pupil intervention too. We feel that, on balance, while teachers were not already 
implementing Mindsets, INSET did not result in a substantially different approach. This explains why 
schools found it easy to implement, but it may also explain why it did not have a strong impact during 
the project's lifetime. 


The role of partners in inplementation 


The project had been attractive to the two external stakeholders, Portsmouth Football Club (PFC) and 
Portsmouth Education Business Partnership (EBP) who delivered additional sessions aimed at 
widening pupils’ horizons. Sessions for the intervention pupils were aimed at reinforcing messages 
delivered by PSAs in the workshops and at encouraging them to adopt a growth mindset in relation to 
their future prospects. Representatives of both partner organisations saw it as fitting well with their 
central remit and existing work. Project leaders in both organizations felt that their staff had benefited 
from the theoretical underpinning which Mindsets gave to their raising aspirations work. One of the 
partners had developed a similar session for other projects with schools, incorporating the Mindsets 
approach and was embedding the approach in its work more generally. They also appreciated the 
development of closer links with the university and with a research project. The project therefore had a 
capacity-building element for the partners involved. 


Fidelity 


NIESR assessed fidelity within the process evaluation largely through interviews with teachers and 
other project participants, through observations of a small sample of pupil workshops and teacher 
INSET and through using the findings of a short survey of control schools carried out by Portsmouth 
University. 


Fidelity of the pupil intervention 


PSAs were used to deliver the intervention in schools to allow it to be tested in a way which allowed for 
the treatment to be delivered blind to teachers. This method meant that teachers were not able to 
compensate for control group pupils by, for example, supporting their learning with a mindset approach 
or providing additional attention and encouragement. It was also intended to trial a cost-effective model 
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of delivery using non-teaching staff. PSAs were employed for around three hours a week and attended 
two training days: the first covered issues of behaviour management in the classroom, ethics, research 
skills, practicalities and administration of assessment sessions; the second training course assigned 
PSAs to either the full Changing Mindsets approach or the study skills only sessions. They were 
assigned to one of these groups and trained separately to avoid contamination through raising 
awareness of Mindsets among the study skills group. Sessions involved training in activities for each of 
the classroom workshops, scripts and supporting materials. The PSAs interviewed for the process 
evaluation said the training prepared them for the intervention, was interesting and engaging. Some of 
the PSAs already knew about mindset theory and practice through their psychology course. 


PSAs in the study skills group were aware that their colleagues were delivering a mindset approach. 
However, they said that they felt it unlikely that they would use the approach within study skills because 
they understood the need to ‘keep to the script’. They also felt that, in any case, they did not know the 
Mindsets techniques in sufficient detail. This suggests that the project had taken necessary steps in 
design and delivery to meet the EEF's requirements for fidelity. 


PSAs had administered tests in some of the schools, reporting that the pupils were very receptive, that 
the process had been straightforward and unproblematic. No issues were apparent which might 
compromise fidelity in this aspect of the project design. Some PSAs had been able to put into practice 
some of the skills they had learned during the training, including in behaviour management. This had 
given them confidence for delivering the intervention. 


Pupil absence meant that all pupils did not attend all workshops. Table 6, based on the data collected 
by the Portsmouth team, summarises the number of workshops attended by pupils in the intervention 
and control groups. For the intervention group 56% of pupils attended all workshops and a further 32% 
attended five out of the six workshops. The equivalent figures for the control (Study skills) group were 
42% and 44% . 


Table 6. Number of workshops attended in the pupil intervention 
Intervention Control 
Number of workshops Frequency Frequency 
attended 


Fidelity of the INSET intervention 


The training delivered in two INSET sessions appeared to provide the necessary conditions for fidelity 
in terms of delivery and to ensure implementation, as far as possible, given that INSET schools were 
not obliged to deliver an intervention as such. 


Teachers found the approach appealing and attractive and readily adopted what they saw as its key 
features. When surveyed, they identified these as: ‘praising the effort rather than the intelligence’ and 
‘seeing mistakes as part of the learning process’. The other two features seen as key, but to a lesser 
extent, were ‘setting high expectations for learning effort, persistence and eventual attainment’ and 
‘celebrating mistakes’. These were, as one might expect, the features they said they had used in 
teaching and learning. Teachers said that all their schools had been praising the effort rather than the 
intelligence and treating mistakes as part of the learning process. Setting high standards for learning 
effort and celebrating mistakes were also reported as having been adopted by the INSET schools. 
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These features were also seen as most useful by teachers for their own practice and, perhaps 
surprisingly, celebrating mistakes was rated less highly. Responses to a range of questions indicate 
that schools found ‘helping pupils to see they can change how intelligent they are’ was seen as less 
useful to participating schools who saw it more difficult to convey this to children and something which 
is an outcome rather than a target. 


When followed up a year after the training, it was apparent that some schools had adapted the Mindsets 
approach, but not in ways which compromised its fidelity. Adaptations included linking mindset to 
emotional intelligence, independent thinking and mind management, and combining it with the Building 
Learning Power (BLP) approach. Most had used it at whole school level, in corridor displays, pupil 
reward systems, school mottos and mission statements, gifted and talented programmes and themed 
assemblies. These adaptations are likely to have reinforced classroom approaches and strengthened 
its impact rather than compromised fidelity. 


Outcomes 


To assess outcomes from the intervention qualitatively, NIESR attended pupil intervention sessions. 
Our observations found that children in both mindset and study skills groups were engaged in the 
activities. Those in the mindset group demonstrated clear understanding of the key concepts of growth 
and fixed mindset and learning by mistakes. This was shown in exercises within the mindset group 
where pupils were asked to assign statements depicting the two types of mindset to descriptions of 
particular characters. The mindset characters used in the materials appeared to assist learning and 
memory recall around some of the key messages. A ‘glossary challenge’ activity showed understanding 
among mindset pupils of key concepts including ‘resilience’, ‘role model’ and _ ‘strategy’. 


Pupils were also able to recall content and key messages of earlier sessions, for example where a 
Mindsets character had failed in their efforts to make a cake, and were also able to recall famous people 
who had not given up when facing significant challenge. We observed some examples of full 
explanations given by pupils, demonstrating detailed understanding on the part of children in the 
Mindsets group. Pupils who were not vocal, in either the mindset or study skills sessions, were 
encouraged to become involved by PSAs assisting the session leader. PSAs from both groups were 
positive about the sessions, their experiences of delivering the programme within the school, and about 
its potential to improve learning. 


Teachers in the INSET schools reported that children had readily understood the mindset theory, helped 
by the clear vocabulary of learning within the project. Teachers felt that the project had equipped 
children with the language of learning, enabling them to verbalise and understand their own learning 
processes. Children’s understanding of the concepts was such that they were able to identify fixed 
mindsets in their classmates. Some teachers said they had seen a difference in pupil performance, but 
felt that the approach was most likely to be effective in the longer rather than short term. There was a 
view that the approach was particularly effective in relation to maths teaching, where pupils were more 
likely to believe they lacked ability and to give up. More widely, schools felt that the approach was 
effective with pupils who feel that they are poor learners, that education is not for them and who tend 
to disengage from learning. Therefore, anumber commented that consistent use is necessary, and that 
it needs to be part of a whole school approach. 


However, teachers felt that, to have an impact, Mindsets would need to be used both consistently and 
in the longer term, starting in the early years. They also felt it should be part of a whole school approach 
and reinforced through regular INSET, cascade learning and shared practice. 


The only barrier to implementation was identified in lack of reinforcement from parents whose fixed 
mindset approach was seen to undermine its impact. One of the INSET schools had organised a 
session for parents to address this, but attendance had been poor. 
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Control group activity 


Portsmouth University gathered information from schools in the control group for the INSET intervention 
in order to assess the extent to which these schools were aware of the project’s approach in ways which 
might influence their practice. Their responses indicate that most teachers in the control schools (72 
out of 126) were not familiar with growth mindset. However, many were familiar with the work of Carol 
Dweck, Guy Claxton (Building Learning Power) or Shirley Clarke (Formative Assessment) (74 out of 
126). This is comparable to schools in the INSET intervention group, where the majority of teachers 
were familiar with the work of these educationalists before the training. The routes by which teachers 
in the control schools had become familiar with the work of these educationalists varied. A small number 
of schools had held training days around the Building Learning Power (BLP) approach of Guy Claxton. 
A number of teachers had attended a cluster meeting on this approach. Other teachers had read books 
by either Carol Dweck or Shirley Clarke, or summaries of their approaches. Some recently qualified 
teachers had become aware of their work within teacher training. One of the control schools had carried 
out an in-school research project. 


Control schools were therefore familiar with some of the theoretical groundings and techniques of the 
growth mindsets approach, in particular through training and reading in the BLP approach, and to a 
lesser extent the work of Shirley Clarke and Carol Dweck. Teachers were also asked about the extent 
to which they had implemented this work in their classroom or school. Although around a quarter of 
teachers (18 out of 74) said they had not yet done so, the others said they had adopted a number of 
features of these approaches to some extent. Specific approaches included using the four Rs of the 
BLP approach—resourcefulness, resilience, reflectiveness, reciprocity—to talk about learning, and the 
‘learning powers’ defined by Guy Claxton. In some cases, these were used as whole school approaches 
as well as by individual teachers. Other teachers were less specific about the aspects of these 
approaches they had adopted, referring to practices such as praising effort not ability, and developing 
a culture of working hard and where mistakes are part of learning. The responses of some teachers 
suggest that some had not applied these principles consistently or rigorously. 


The responses of control schools to this questionnaire do suggest that ‘business as usual in the control 
schools was not an absence of mindset-related approaches. This was partly because of existing 
projects and activity of educationalists within the Portsmouth and Hampshire areas, and because such 
approaches often form part of teacher CPD courses even if not specifically focused on mindset. The 
pupil intervention is also likely to have raised awareness and encouraged interest among teachers who 
may have started to use mindset approaches within their classes and therefore with control group 
pupils. As we explain in the next section, the general awareness of growth mindset as an approach to 
teaching and learning is likely to have weakened the impact of both the pupil and INSET interventions 
and may have contributed to the absence of a measurable effect. 
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Conclusion 


Key conclusions 


1. Pupils who received the growth mindset workshops made an average of two additional months’ 
progress in English and maths. These findings were not statistically significant, which means 
that we cannot be confident that they did not occur by chance. However, the finding for English 
was Close to statistical significance, and this suggests evidence of promise. 

2. Pupils whose teachers received the professional development intervention made no additional 
progress in maths compared to pupils in the control group. These pupils made less progress in 


English than the control group, but this finding is not statistically significant and we cannot be sure 
that it did not occur by chance. 


3. FSM-eligible pupils who were involved in the professional development intervention gained a better 
understanding of the malleability of intelligence. 

4. Intervention and control schools were already using some aspects of the growth mindsets 
approach. This may have weakened any impact of the interventions. 


5, Future trials could examine the impact of a programme that combines the two interventions and 
runs for a longer period of time. 


Limitations 


The pupils in the intervention group for the INSET trial were more likely to be eligible for free school 
meals (FSM) and they also scored lower on the pre-tests, indicating an imbalance between the 
intervention and control groups. This was despite finding balance between the two groups based on 
similar characteristics for an earlier year group that was used as the basis for randomising the schools. 
These differences were controlled for in the analysis. No such issues arose with the pupil intervention. 


The findings of the process evaluation also suggest two sources of bias: the extent to which schools 
were already using aspects of the Mindsets approach, and the use of mindset-related approaches in 
some of the control schools. We believe that pre-exposure to and use of some of the components of 
the approach may have weakened its potential impact. This and other possible explanations for the 
absence of a statistically significant effect are discussed below. 


Interpretation 


The findings of the process evaluation suggest that the potential impact of the Changing Mindsets 
interventions on pupil test scores may have been weakened by four factors: 


e use of some features of the mindset approach in test schools prior to the project; 
e use of some mindset-related approaches in many of the control schools; 

e the duration and intensity of the pupil intervention; and 

e the duration and scope of the INSET intervention. 


The process evaluation provided evidence that schools in both the treatment and control groups were 
already using aspects of the mindset approach. The responses of some teachers suggest that these 
had not been applied consistently or rigorously before training, but were applied consistently following 
the INSET. A number of teachers commented that consistent use is necessary for the Mindset approach 
to be effective, and that it also needs to be part of a whole-school approach. However, while this is 
likely to be true, it is possible that pre-exposure to and use of some of the components of the approach 
may have weakened the impact of both the pupil and INSET interventions and may have contributed to 
the absence of a measurable effect. 


Another potential explanation for the absence of a statistically significant impact is the duration and 
intensity of the intervention. The pupil workshops intervention consisted of six reasonably intensive 
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sessions followed by sessions from external partners in the community. The additional sessions 
delivered by external partners did include a focus on developing a Growth Mindset, but this had the 
intention of reinforcing messages rather than providing the same level of focus on mindsets as the 
sessions that took place in the school. While this reinforcement is likely to have been valuable, the 
teachers of the pupils in the pupil workshop intervention were not part of the project, so the approach 
and messages of mindset will not have been reinforced in the classroom or in daily school life. It is 
possible that the intervention was not intensive or sustained enough to have made a statistically 
significant difference to pupils’ academic achievement. Similarly, the training delivered to the schools 
involved in the INSET intervention may have been insufficient in length or intensity to have made a 
statistically significant impact on pupils’ attainment. 


A final observation, made by some teachers, was that while they had already seen a difference in pupil 
performance, change was a long-term process which involved continual reinforcement of mindset 
principles. Teachers felt that, to have an impact, a growth mindsets approach would need to be used 
consistently for a longer period of time, starting in the early years and reinforced in the home. They also 
felt it should be part of a whole-school approach and reinforced through regular INSET, cascade 
learning and shared practice. This seems like a reasonable assessment of the potential of growth 
mindsets theory to have an impact on pupil outcomes. 


Future implementation of Mindsets 


Findings from the process evaluation highlight some key issues relevant to future use of Mindsets in 
UK schools, including EEF target schools and pupils. These concern: 


e the appeal of mindset approaches to teachers; 

e the ‘fit’ of mindset with schools’ ethos; 

e its relative simplicity and ease of use; and 

e understanding of the approach by pupils and, potentially, parents. 


The evaluation did not find a statistically significant effect on attainment in either maths or English. 
However, as we have stated, this may be accounted for by contextual factors. Pre-exposure of many 
teachers to mindset principles and their application in control schools is likely to have reduced its impact. 
Ironically, the project's impact may have been weakened by growing popularity and prevalence of 
mindset-type theory and practice. This is exemplified, for example, in debates about resilience and risk- 
taking. The intervention was also relatively short, at only six intensive sessions for the pupil impact, 
followed by four sessions or visits by project partners, and a relatively short period of training for 
teachers in the INSET intervention. 


The absence of a statistically significant effect should not be taken to mean that the project did not have 
impact. It clearly influenced the thinking and practice of participants, including teachers, PSAs and 
project partners at Portsmouth FC and the Education Business Partnership. We have described how 
the project was well regarded by its participants, particularly teachers in the INSET intervention and the 
project partners hosting the visits. The training and the resources were seen as useful, and had enabled 
teachers to incorporate a mindset approach into day-to-day teaching and whole school activities. 
Importantly, the approach was easy to implement and was seen to have a close fit with the ethos of 
participating schools. It is therefore likely to be equally suitable for use in other EEF target schools. We 
would suggest that to be effective, it should be used consistently and over a longer period than the 
project allowed. 


Schools reported that children had readily understood the mindset theory, helped by the clear 
vocabulary of learning within the project. Our observations confirm these reports. Some teachers said 
they had seen a difference in pupil performance, but felt that the approach was most likely to be effective 
in the longer rather than short term. There was a view that the approach was particularly effective in 
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relation to maths teaching, where pupils were more likely to believe they lacked ability and to give up. 
More widely, schools felt that the approach was effective with pupils who feel that they are poor learners, 
that education is not for them and who tend to disengage from learning. 


Implications for future EEF projects 


The project also contains some further learning points for future EEF projects. These include the use 
of project support assistants (PSAs) to deliver the pupil intervention in schools and to administer the 
pupil tests. The use of university students interested in a teaching career in this role was cost-effective 
and efficient. It also had wider benefits in giving students the opportunity to acquire classroom 
experience for applying to teacher training. The project may therefore have had a capacity-building 
aspect which might be incorporated into future EEF projects. A second additional feature of the project's 
design is its use of project partners, in this case Portsmouth FC and the Education Business 
Partnership. Their involvement undoubtedly added to the experience of children in the pupil intervention 
but also increased the capacity of these organisations to work alongside schools to improve pupil 
outcomes. 
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Appendix 1: Pupil ere 3@ ‘ letter 


Portsmouth — University of 
CITY COUNCIL Portsmouth 


Department of 
Psychology 
King Henry Building 


5'* December 2012 


Changing mindsets and study skills workshops: enhancing children’s attainment 


Dear Year 5 Parent 


Your child’s school is taking part in a project to see if different approaches to teaching learning skills 
improve children’s educational attainment. We are writing to tell you more about what this project entails 
for your child. 


The project team 

This project is an extremely exciting opportunity to increase pupil attainment and is funded by the 
Education Endowment Foundation (EEF, a government-supported charity). It is being run as a 
partnership between the Department of Psychology at the University of Portsmouth (UoP), Portsmouth 
City Council (PCC) and schools around Hampshire. As part of this project we are also working with a 
team of researchers at the National Institute of Economic and Social Research (NIESR) who have been 
recruited by the EEF to evaluate the success of the project. 


The project 

In summary, a team from the University of Portsmouth will be visiting your child’s schools for six weeks 
to deliver a half-day session each week as part of normal school classes (in groups of about 15 pupils). 
Four additional sessions (over four more weeks) will be delivered by Pompey in the Community and the 
Education Business Partnership team. 


All learning skills sessions will focus on spelling and maths strategies, as well as activities on planning, 
goal setting, managing time and working effectively; however one will also focus on helping children to 
see intelligence as developable. All sessions have structured lesson plans and materials that make the 
lessons fun, child-friendly and age appropriate, e.g. mind mapping, board games, specially designed 
comic strips. Learning will be active and aim to build learning skills and confidence. We expect both 
types of workshop to have a positive impact on the pupils. However, at the end of the project all teachers 
in your school will be offered teacher training so that they can support all children in whichever approach 
proved to be most successful. 


Evaluation of the project 

In order to check whether the learning skills sessions are helping pupils’ attainment, we will be 
assessing their numeracy and literacy (using standardised attainment tests commonly used in schools). 
We will assess attainment three times over the next two years (starting in January 2013). At the same 
time we will measure children’s views about their future life and their approach to learning. These 
assessments require your child to respond to items, saying how much they agree or disagree with them. 
For example, ‘I have enough time to finish the things | really want to do’ (belief about future life) and 
‘making mistakes is a really good way to learn’ (approach to learning). Each assessment will last 1 hour 
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and 30 minutes including a 15-minute break. A teacher or support teacher will be present during these 
sessions and all staff from the University of Portsmouth working on the project are CRB checked. 


Information about your child 

The information that we collect in the assessments described above will contain pupils’ names in the 
early stages of data collection (but in accordance with the Data Protection Act this will be securely 
transported to the University and will remain confidential). It will be stored in a locked cabinet until the 
data is entered onto an encrypted memory stick, at which point children’s names will be replaced with 
a unique identifying number. Some additional information will be provided by Portsmouth and 
Southampton City Councils under their data sharing agreement (e.g. free school meal provision, prior 
attainment data). Your school’s management team will have access to the attainment results collected 
by the project team only for pupils in their school. PCC and the research project team at the University 
of Portsmouth will have ownership of the data. The project evaluators (NIESR) and the EEF will have 
access to it without pupil names attached. The Department of Psychology will keep raw data which has 
no pupil names linked to it. This will be stored in a locked archive room in the department for five years 
after any publications associated with it. After this period, all information about your child will be 
destroyed. Please contact us if you have any concerns about this data sharing. 


Keeping you informed 

After each assessment we will send a note home to you via your child in order that you are aware of 
when a testing session has taken place, giving you the chance to discuss it with your child. Your child 
will also be given a verbal briefing and debriefing to help them understand why they were being 
assessed. They will be told that we are not interested in their attainment but the attainment of the group, 
to see if our learning skills sessions have helped them. 


Please do not hesitate to contact your school or the project team (contact details below) with any 
questions that you have. 


Yours faithfully 


’ 


ioe 


Dr Sherria Hoskins 

Project Lead at the University of Portsmouth 
Phone: 023 9284 6315 

Email: changingmindsets@port.ac.uk 


Education 
Endowment 
Foundation 
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Appendix 2: Teacher intervention, parent etter } 2 
University of 
Portsmouth 


Portsmouth 
CITY €CoOUuUNCILE 


Department of 


Psychology 
King Henry Building 


5 December 2012 


Changing Mindsets: Assessing teacher training aimed at enhancing children’s learning skills. 


Dear Year 5 Parent 


Your child’s school is taking part in a project to see if training teachers in Mindsets (what we believe 
about intelligence) can improve children’s educational attainment. We are writing to tell you more about 
what this project entails for your child. 


The project team 

Changing Mindsets is an extremely exciting project funded by the Education Endowment Foundation 
(EEF, a government-supported charity) and is being run as a partnership between the Department of 
Psychology at the University of Portsmouth (UoP), Portsmouth City Council (PCC) and schools around 
Hampshire. As part of this project we are also working with a team of researchers at the National 
Institute of Economic and Social Research (NIESR) who have been recruited by the EEF to evaluate 
the success of the project. 


Evaluation of the project 

In order to check whether the teacher training is helping pupils’ attainment, we will be assessing their 
numeracy and literacy (using standardised attainment tests commonly used in schools). We will assess 
attainment three times over the next two years (starting in January 2013). At the same time we will 
measure children’s views about their future life and their approach to learning. These assessments 
require your child to respond to items, saying how much they agree or disagree with them. For example, 
‘| have enough time to finish the things | really want to do’ (belief about future life) and ‘making mistakes 
is a really good way to learn’ (approach to learning). Each assessment will last 1 hour and 30 minutes 
including a 15-minute break. A teacher or support teacher will be present during these sessions and all 
staff from the University of Portsmouth working on the project are CRB checked. 


Information about your child 

The information that we collect in the assessments described above will contain pupils’ names in the 
early stages of data collection (but in accordance with the Data Protection Act this will be securely 
transported to the University and will remain confidential). It will be stored in a locked cabinet until the 
data is entered onto an encrypted memory stick, at which point children’s names will be replaced with 
a unique identifying number. Some additional information will be provided by Portsmouth and 
Southampton City Councils under their data sharing agreement (e.g. free school meal provision, prior 
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attainment data). Your school’s management team will have access to the attainment results collected 
by the project team only for pupils in their school. PCC and the research project team at the University 
of Portsmouth will have ownership of the data. The project evaluators (NIESR) and the EEF will have 
access to it without pupil names attached. The Department of Psychology will keep raw data which has 
no pupil names linked to it. This will be stored in a locked archive room in the department for five years 
after any publications associated with it. After this period, all information about your child will be 
destroyed. Please contact us if you have any concerns about this data sharing. 


Keeping you informed 

After each assessment we will send a note home to you via your child in order that you are aware of 
when a testing session has taken place, giving you the chance to discuss it with your child. Your child 
will also be given a verbal briefing and debriefing to help them understand why they were being 
assessed. They will be told that we are not interested in their attainment but the attainment of the group, 
to see if our teacher training has helped them. 


Please do not hesitate to contact your school or the project team (contact details below) with any 
questions that you have. 


Yours faithfully 


iy 


\Proe 


Dr Sherria Hoskins 

Project Lead at the University of Portsmouth 
Phone: 023 9284 6315 

Email: changingmindsets@port.ac.uk 
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University of 
Portsmouth 


Appendix 3: Opt-out form 


20 June 2014 


Assessing the Changing Mindsets Project - working with schools to help children succeed at 
school 


Dear Year 6 Parent 
Your school has kindly sent this letter out to you on our behalf. 


You may remember the letter that we sent out to you early in December 2012 about the Changing 
Mindsets project that your child’s school took part in when your child was in Year 5. This was to see if 
training teachers and working with pupils around effort and persistence would improve how well they 
do at school. This project involved 36 local schools and was run as a partnership between the 
Department of Psychology at the University of Portsmouth (UoP) and your local authority. The project 
is funded by the Education Endowment Foundation, a charity dedicated to ensuring that children from 
all backgrounds can fulfil their potential. 


We are now writing to ask your permission for the project team (the University of Portsmouth) and the 
project evaluators (the National Institute for Social and Economic Research) to obtain information about 
your child from the records held in the Department for Education’s National Pupil Database and match 
this to the information that we collected during the project. The information that we intend to obtain from 
the Department for Education’s National Pupil Database is what is called tier one data; this includes 
(but is not limited to) information such as ethnicity, gender and free school meal details. This information 
is required in order for us to explore whether the training that we have offered has helped pupils and if 
so which groups of pupils. This will enable us to improve training for the future. 


The information that we receive from the Department for Education National Pupil Database will not 
have pupil names attached to it, only the Unique Learner Number. This is a number that schools and 
the Department for Education use to keep information confidential. In accordance with the Data 
Protection Act all information that is given to us will be securely transported and stored. This data will 
be released to the research, evaluation and funding team only for the purpose of evaluating the project. 
This data will not be used for any other purpose. Any information published about the project will be 
done in such a way as not to identify any child or school. Ten years after the project is complete 
(September 2024), all data pertaining to this research will be securely destroyed. 


If you are content to allow us to progress you do not need to do anything. If you would like to opt your 
child out, so that we do not request this data from the National Pupil Database, please contact us by 
Friday 4" July 2014. You can do this by phoning, emailing or writing to us using the details below. The 
University of Portsmouth team will then provide the names of the children who have not been opted out, 
to the Local Authority who will provide us with the relevant Unique Learner Numbers. 


Please contact us with any further questions that you have. 
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Yours faithfully 
Dr Sherria Hoskins, Project Lead at the University of Portsmouth 


To opt out please contact us to tell us your child’s name and the school they attend by: 
phoning the Changing Mindsets project team at the University of Portsmouth on 023 9284 6315 
or email us at changingmindsets@port.ac.uk 

or write to us at: 


The Changing Mindsets Team 

Department of Psychology, University of Portsmouth 
King Henry Building, King Henry | Street 
Portsmouth, Hampshire, PO1 2DY 
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Appendix 4: Full estimation results 


Table A1a: Full estimation results, pupil intervention 
Maths English WY TTate FX=3 


[1.098] [0.654] [0.588] 
[11.396] [7.056] [6.248] 
Age squared 0.049 0.015 -0.030 
[0.050] [0.031] [0.027] 
[1.055] [0.647] [0.566] 


[0.284]* [0.178] [0.172] 

[0.080]** [0.064] [0.056] 
EAL share missing 2.475 -0.397 1.438 

[1.774] [1.522] [1.119] 


[0.054]** [0.057]** [0.095]** 
Constant 650.075 192.140 -388.857 


* p<0.05; ** p<0.01 


SEN share inschool = -0.651 0.288 -0.121 


[1.276] [0.738] [0.619] 
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Table A1b: FSM sample estimation results, pupil intervention 
Maths English WY TTate FX=3 


[1.631] [1.158] [0.866] 
[18.195] [13.042] [9.935] 
[0.080] [0.057] [0.044] 


Female 0.096 0.814 -0.779 


[0.123}* [0.107] [0.086] 


EAL share missing 4.677 4.826 1.534 


[2.823] [2.520] [1.554] 
Pre-test 1.253 0.650 0.347 

(0.111) [0.082]** [0.173] 

[1,038.025] [745.666] [564.502] 


R? 0.79 0.63 0.16 


N 61 64 62 
5 5 5 


* p<0.05; ** p<0.01 
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Table Aic: Low pre-test sample estimation results, pupil intervention 
Maths English Tate X=3 
[1.408] [1.325] [1.177] 


[15.302] [13.297] [14.423] 
Age squared -0.001 -0.024 -0.003 
[0.067] [0.058] [0.063] 


Female 0.851 0.792 1.223 
[1.464] [1.207] [0.954] 

[0.521] [0.463] [0.353] 

[0.120]* [0.150] [0.096] 

EAL share missing 3.831 6.187 3.013 
[3.566] [3.589] [2.089] 

[0.340]* [0.195]** [0.600] 

Constant -19.365 -304.289 -22.528 
[873.955] [757.768] [820.459] 


N 55 57 52 
N schools 5 5 5 


* p<0.05; ** p<0.01 
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Table A1d: Full estimation results, INSET intervention 
Maths English Mindset 


impact 0.176 -0.685 -1.040 
[1.330] [0.544] [0.681] 


Age (months) -10.350 -3.833 1.440 
[5.824] [2.750] [2.519] 
[0.025] [0.012] [0.011] 
Female 0.591 0.485 0.283 


[0.119] [0.048] [0.062] 

[0.047] [0.019] [0.024]* 
EAL share missing -0.663 -0.609 -1.428 
[2.171] [0.883] [1.122] 

[0.034]** [0.020]** [0.036]** 


Blocking variables 


Constant 602.119 224.589 -72.433 
[333.253] [157.331] [144.173] 


Log likelihood -3161.777 -2463.547 -2369.299 
aly -squared test of 17.70 7.79 27.88 


ch -Squared p-value 0.00 0.003 0.00 
896 885 878 
arr effects 
School level 4.711 0.657 1.440 
variance 
[2.423] [0.428] [0.686] 
Pupil-level 66.719 14.757 12.291 
variance 
[3.202] [O. ial [0.597] 


N observations per 
school 


min per school 14 14 14 


max per school 105 104 101 
mean per school 37 37 37 
Icc 0.065 0.043 0.105 


* p<0.05; ** p<0.01 
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Table Ale: FSM sample estimation results, INSET intervention 
Maths English Tate X=) 


Impact 0.527 -0.041 -3.067 
[1.695] [0.790] [0.774]** 


Age (months) -15.298 -11.652 2.878 
[13.298] [6.870] [5.814] 

[0.058] [0.030] [0.025] 

Female 1.273 0.721 0.448 


[0.062] [0.028] [0.028] 

[2.924] [1.390] [1.352] 
Pre-test 1.204 0.697 0.291 

[0.081]** [0.045]** [0.080]** 
Blocking variables Yes Yes Yes 
Constant 871.607 666.584 -147.026 

[761.482] [393.353] [332.978] 

Log likelihood -612.529 -497.654 -470.971 
aly -squared test of 0.34 0.000 0.60 


ch -squared p-value 0.279 1.00 0.220 
178 176 176 
ea effects 


School level 2.057 0.000 0. 557 
variance 


[4.023] [0.000] [O. 860] 
Pupil-level 62.381 16.919 12.020 
variance 


ie 130] ie 874] a 383] 


N schools _ 
school 


max per school 14 14 14 
mean per school 8 8 8 


Icc 0.032 0.000 0.044 
* p<0.05; ** p<0.01 
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Table A1f: Low pre-test sample estimation results, INSET intervention 
Maths English Tate X=) 


Impact 1.713 -0.130 -1.670 
[1.394] [0.905] [0.987] 
Age (months) -2.467 -9.298 11.094 


[9.862] [4.639]* [5.091]* 


Age squared 0.011 0.040 -0.049 
[0.043] [0.020}* [0.022]* 


Female 1.905 0.385 -0.276 
[0.945]* [0.457] [0.479] 
[0.117] [0.079] [0.079] 
[0.054] [0.033] [0.035] 

EAL share missing -1.028 -0.609 -2.174 
[2.199] [1.473] [1.479] 
[0.287]** [0.067]** [0.216] 

Blocking variables Yes Yes Yes 

[563.528] [265.035] [291.312] 


Chi-squared test of 0.55 7.52 5.16 


RE 
N 256 292 225 
- School level 1.671 1.779 1.721 
variance 
2.741 [1.160] [1.233] 
- Pupil-level 53.996 13.773 11.955 
variance 


[5.050] [1.200] [1.203] 
N schools 24 24 23 


N observations per 
school 
min per school 2 1 2 


max per school 26 31 27 


mean per school 11 12 10 


Icc 0.030 0.114 0.126 
* p<0.05; ** p<0.01 
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Appendix 5: Security classification of trial findings 


P ‘ _Th 
1.Design: 2. Power 3. Attrition: 4. Balance: SALLE 


What is the elie What is the validity: 


Adjustment 


: minimum ; Adjustment to 
quality of the (AV7-) Me) ie | ge) to rating to ; 
detectable account for issues 
design of the out from the account for 
; effect at the with 
evaluation? evaluation? balance? 
start? interpretation? 
Evaluation design Implementation Analysis and interpretation 


Rating 1. Design 3. Attrition 4. Balance oY Malaeyeleom ce) 
(MDES) validity 


Well-balanced on ois 
Fair and clear experimental 
design (RCT, RDD) < 0.3 


Well-matched comparison 
(quasi-experiment) 
Matched comparison 
(quasi-experiment) 
Comparison group with 

poor or no matching < 0.6 


The padlock ratings for the INSET intervention effects are shown in light green, and pupil intervention 
effects in dark green. 


Light — INSET: MDES 0.3-0.4; 20% attrition; some indication of imbalance in EAL and level-4, but not 
enough to be a problem probably; blinded data collection. 


Dark — Pupil: MDES 0.4-0.5; 39% attrition; little imbalance; blinded data collection. 


The final security rating for this trial is 3 @ for the INSET intervention effects, and 2 @ for the pupil 
intervention effect. This means that the conclusions about INSET have moderate security. 


This evaluation was designed as a randomised controlled trail. The sample size was designed to 
detect a MDES of less than 0.4, by design, reducing the security rating to 3 @. At the unit of 
randomisation (school), there was zero attrition, and extremely low attrition at the pupil level also. 
The post-tests were administered by the schools by teachers who were aware of the treatment 
allocation. Balance at baseline was high, and there were no substantial threats to validity. 
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Appendix 6: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. More information about the EEF’s approach to cost evaluation can be found on the 
EEF website. Cost ratings are awarded as follows: 


Cost rating Description 


£ Very low: less than £80 per pupil per year. 
££ Low: up to about £200 per pupil per year. 


EEE Moderate: up to about £700 per pupil per 
year. 


ELEEEL High: up to £1,200 per pupil per year. 


ELEELEELE Very high: over £1,200 per pupil per year. 
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